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Abstract — We consider Slepian-Wolf code design based on 
LDPC (low-density parity-check) coset codes for memoryless 
source-side information pairs. A density evolution formula, 
equipped with a concentration theorem, is derived for Slepian- 
Wolf coding based on LDPC coset codes. As a consequence, an 
intimate connection between Slepian-Wolf coding and channel 
coding is established. Specifically we show that, under density 
evolution, design of binary LDPC coset codes for Slepian- 
Wolf coding of an arbitrary memoryless source-side information 
pair reduces to design of binary LDPC codes for binary-input 
output-symmetric channels without loss of optimality. With this 
connection, many classic results in channel coding can be easily 
translated into the Slepian-Wolf setting. 

I. Introduction 

Consider the problem of encoding X with side information 
Y at the decoder. Here {X, Y) are two memoryless sources 
with joint probability distribution P{x,y) on X x y. This is 
a special case of Slepian-Wolf coding. General cases can be 
reduced to this special case via either time-sharing or source 
splitting. For this special case, the Slepian-Wolf theorem states 
that the minimum rate for reconstructing X is H{X\Y). For 
simplicity, we assume X = {0, 1} and |3^| < cx) throughout 
this paper The general finite-alphabet case can be reduced 
to this special case via multilevel coding. In the literature 
many low-complexity Slepian-Wolf coding schemes have been 
proposed [2]-[5], almost all of which are based on ideas from 
channel coding, and simply use some binary linear channel 
codes as Slepian-Wolf codes. In this approach it is implicitly 
assumed that a good channel code for a channel is also a 
good Slepian-Wolf code for the same channel linking the 
source and the side information; yet few justifications have 
been presented except for [6]. Motivated by these observations, 
in this paper we consider Slepian-Wolf code design based on 
binary LDPC coset codes (see Section Hll for details). To this 
end we derive the density evolution formula for Slepian-Wolf 
coding, equipped with a concentration theorem. An intimate 
connection between Slepian-Wolf coding and channel coding 
is then established. Specifically we show that, under density 
evolution, any Slepian-Wolf source coding problem, where the 
joint distribution P{x,y) can be arbitrary, is equivalent to a 
channel coding problem for a binary-input output-symmetric 
channel. Note that this channel is often different from the 
channel between the source and the side information in the 
original Slepian-Wolf coding problem. This is in sharp contrast 



to the practice in the works reviewed above where the two 
channels are assumed the same. 

The rest of this paper is organized as follows. In Section 
mi we review some basic results in the channel coding the- 
ory. The emphasis will be on binary linear codes with the 
practical belief-propagation decoding algorithm. In Section 
Unl we develop the belief-propagation algorithm for Slepian- 
Wolf coding. The associated density evolution formula and 
concentration theorem are also provided. An intimate connec- 
tion between Slepian-Wolf coding and channel coding under 
density evolution is established in Section II VI We conclude 
the paper in Section 

II. Review of Channel Coding Theory 

Any binary linear code C = {c} can be expressed as the set 
of solutions c to a parity check equation He = 0, where H is 
called the parity check matrix of C, and here multiplication 
and addition are modulo 2. Given some general syndrome 
s G {0,1}"^'"', the set of all n-length vectors x satisfying 
i/x = s is called a coset Cs- A dc) -regular low-density- 
parity-check (LDPC) code is a binary linear code determined 
by the condition that every codeword bit participates in exactly 
dy parity-check equations and that every such check equation 
involves exactly dc codeword bits. Given the parity check 
matrix H, we can construct a bipartite graph with n variable 
nodes and m — n — k check nodes. Each variable node 
corresponds to one bit of the codeword, and each check node 
corresponds to one parity-check equation. Edges in the graph 
connect variable nodes to check nodes and are in one-to-one 
correspondence with the nonzero entries of H. The ensemble 
C'"'{dv,dc) of (d^, dc)-regular LDPC codes of length n is 
defined in [7]. We can also define the irregular code ensemble 
C"(A, p), where {X, p) denotes a degree distribution pair [7]. 

The belief-propagation algorithm is an iterative message- 
passing algorithm. Let my} denote the message sent from 
variable node v to its incident check node c in the ^th iteration. 
Similarly, let rnS denote the message sent from check node 
c to its incident variable node v in the lih iteration. The 
update equations for the messages under belief propagation 
are described below: 
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where C„ is the set of check nodes incident to variable node 

V, Vc is the set of variable nodes incident to check node c, 
mo = In p|^'j^'~'-^^j is the initial message associated with the 
variable node v, and 7(0;) = (sgn(a::), — Intanh |||). 

The performance of LDPC codes under the belief- 
propagation algorithm is relatively well-understood for binary- 
input output-symmetric (BIOS) channels. 

Definition 1: A binary input channel with transition proba- 
bility function p{-\-) : X xy ^ [0, 1] from A" to 3^ is output- 
symmetric if there exists an injective map : y ^ M. such 
that 

pivivt) = v{y)\xt = o)=p{y^{yt) = -v{y)\xt = i),Vy e y, 

where yt is the channel output in response to channel input 

Xt- 

An important property of the BIOS channel is that under the 
belief-propagation algorithm, the decoding error probability is 
independent of the transmitted codeword. So without loss of 
generality, we can assume the all-zero codeword is transmitted. 

In order to analyze the asymptotic (in codeword length) 
performance of the LDPC code ensemble C"(A, p), a powerful 
technique called density evolution is developed in [8], [9]. The 
iterative density evolution formula for BIOS channels is given 
in the following theorem. 

Theorem 1 ( [9], Theorem 2): For a given BIOS memory- 
less channel let P^"^ denote the initial message density of 
log-likelihood ratios, assuming that the all-zero codeword was 
transmitted. If, for a fixed degree distribution pair (A, p), 
P*^'' denotes the density of the messages passed from the 
variable nodes to the check nodes at the /th iteration of behef 
propagation then, under the independence assumption 

p(i) = p(o) ^ A(r-i(p(r(p('-i))))) (1) 

where F is the density transformation operator induced by 7. 
The theorem below provides a theoretical foundation of den- 
sity evolution. 

Theorem 2 ( [8], Theorem 2): Over the probability space 
of all graphs C"{dy,dc) and channel realizations let Z be the 
number of incorrect messages among all nd^ variable-to-check 
node messages passed at iteration I. Let pe^ be the expected 
number of incorrect messages passed along an edge with a 
tree-like directed neighborhood of depth at least 21 at the Ith 
iteration, i.e., 
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Then, there exist positive constants (3 = P{dy,dc,l) and 7 = 
j{dy,dc, I) such that for any e > and n > ^ we have 

Pr{|Z - ndypl!^\ > nd^e} < 26"''''". 
The main contribution of this paper is that we prove a simi- 
lar density evolution formula with an associated concentration 



theorem for Slepian-Wolf coding, and establish an intimate 
connection between Slepian-Wolf coding and channel coding 
under density evolution. 

For a given joint distribution P(x, y) on X x y, we 
can define two channels: one from X to y with transition 
probability P{y\x), the other from y to X with transition 
probabihty P{x\y). Both channels are somehow related to 
Slepian-Wolf coding as we shall discuss in the following two 
examples: 

Example 1 ( Channel from Y to X ): Suppose X = Y (B Z, 
where X,Y and Z aU assume values in {0,1}, © is the 
modulo-2 addition, and Z ~ Ber(g) (with < g < 5) is 
independent of Y . Let H be an (n, k) binary parity-check 
matrix. Let C (i.e., Co) be the hnear code with the parity 
check matrix H. Assuming that all rows of H are linearly 
independent, there are 2'^ codewords in C, so the code rate is 
(log|C|)/n = fc/n. 

The following scheme was suggested by Wyner [6]. Given 
X, the encoder sends the syndrome s = Hsi to the decoder. 
With the side information y, the decoder can compute Hz, = 
Hx(BHy = s®Hy. Syndrome decoding can be implemented 
to find the minimum weight z such that Hz = Hz. The 
decoder then claims that x = y -(- z is the target sequence x. 
It can be shown that if the error probability of C when used 
over the chamiel from Y to X (i.e., a binary symmetric channel 
with crossover probability q) under syndrome decoding is e, 
then the above Slepian-Wolf coding scheme is also of error 
probability e. Furthermore, if C is capacity-achieving for the 
channel from Y to X, i.e, the rate of C is 1 — H{q), then 
the rate of this Slepian-Wolf coding scheme is H{q), which 
is exactly the Slepian-Wolf limit. 

Example 2 (Channel from X to Y): Suppose X is uni- 
formly distributed over {0, 1} and P{y\x) is a BIOS chaimel. 

The encoding procedure is the same as that in Example 1. 
Given x, the encoder finds the coset Cg that contains x and 
send the syndrome s to the decoder. So the encoder rate is 
(n — k)/n. Given the side information y, the decoder tries 
to recover x using the belief propagation decoding algorithm 
for Cs. For a BIOS channel, the decoding error probability is 
the same for all coset codes Cs under the belief-propagation 
algorithm. So if Cq is a hnear code for channel P{y\x) with 
error probability e under belief-propagation decoding, then 
the error probability of the above Slepian-Wolf coset coding 
scheme is also e. Furthermore, assuming Cq is a capacity 
achieving linear code for channel P{y\x), the above coding 
scheme is then of rate 1 — C = H{X\Y), which is exactly the 
Slepian-Wolf hmit. Here C is the capacity of channel P{y\x). 

When X is nonuniform, we can still use the above coset 
coding scheme as long as Cq is a good channel code for chan- 
nel P{y\x) under the behef-propagation decoding algorithm. 
The reason is that for a BIOS channel, the error probability 
resulting from the belief propagation decoding algorithm is the 
same for every codeword in every coset. Nonetheless, since 
1 — C > H{X\Y) when X is nonuniform, we see that the 
above coset coding scheme fails to achieve the Slepian-Wolf 
limit even when Cq is an optimal channel code for channel 



P{x\y). This phenomenon has been observed in [10]. 

When P{y\x) is not output-symmetric, the decoding er- 
ror probabiHty, under the beHef-propagation algorithm, is in 
general different for different codewords in each coset, and 
also different for different cosets. In this case, the connection 
between channel coding for channel P{y\x) and Slepian-Wolf 
coding is not clear 

We have seen that although the above two examples exhibit 
some interesting connections between channel coding (either 
for the channel from X to y or the channel from Y to X) 
and Slepian-Wolf coding, both of them have severe limitations. 
In this paper we shall provide a general framework, which 
includes these two examples as special cases, and within the 
framework establish the connection between channel coding 
and Slepian-Wolf coding. It should be emphasized that in our 
framework, P{x) does not need to be uniform, and P{y\x) 
does not need to be output-symmetric. 

III. Belief-Propagation Algorithm, Density 
Evolution and Concentration Theorem 

We use the same encoding method as that in Example 1 and 
Example 2. We first fix a parity check matrix H. Given x, the 
encoder sends the syndrome s — Hx to the decoder. But we 
do not use the channel decoding method in Example 2. The 
reason is that in channel coding, codewords are assumed to be 
equally probable, but in Slepian-Wolf coding, codewords are 
generated by P{x), and are in general not equally probable 
if X is not uniform over {0, 1}. It turns out that it is easy 
to incorporate the prior distribution P{x) into the belief- 
propagation. The update equations for the messages in this 
Slepian-Wolf decoding are described below: 
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where mo = In p^^' °!^'| = In V'^^'n , and s is the 
syndrome value associated with check node c. It can be veri- 
fied that this algorithm produces the exact symbol-by-symbol 
a posteriori estimation of x given y when the underlying 
bipartite graph is a tree. We can see that the only difference 
from the channel decoding case is the definition of initial 
message mo. This decoding scheme can be viewed as a MAP- 
version of the belief propagation algorithm, while the channel 
decoding scheme can be viewed as a ML-version of the belief 
propagation algorithm. 

Now we proceed to develop the density evolution formula 
for this belief -propagation algorithm. We use the standard tree 
assumption. Let P^'^ {x) {x = 0, 1) be the message distribution 
from a variable node to a check node at the ^th iteration 
conditioned on the event that the variable value is x. Similarly, 
let Q(')(a;) {x = 0,1) be the message distribution from a 
check node to a variable node at the ^th iteration conditioned 




(2) 



on the event that the target variable value is x. Assume 
P{X = 0) =p. Let (PW) =pP(')(0) + (l-j3)P(')(l)o/-i, 
where /(m) = — m is a parity reversing function. We have 



where pi^ is the expected number of incorrect messages sent 
from a variable node at the Ixh iteration. 

The following theorem provides a density-evolution formula 
for (P«). 

Theorem 3: Under the tree assumption, 

(p(0) = (p(o)) ^ A(r-i(p(r((p('-i)))))). (4) 

Remark: Theorem |3] does not directly follow from the 
approach in [9] since the all-zero codeword assumption is not 
valid in our setting. 

Theorem 4 (Concentration Theorem): Over the probability 
space of all graphs C^{d^, dc) and source realizations let Z be 
the number of incorrect messages among all ndy variable-to- 
check node messages passed at iteration I. Then, there exist 
positive constants (i = f]{dy,dc,l) and 7 = j{dv,dc,l) such 
that for any e > and n > ^ we have 
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Pi{\Z ~ ndypi^^l > nd^e} < 2e" 



It can be verified that the density evolution formula 
and concentration theorem (Theorem |3 do not depend on 
the definition of the initial message, i.e., they still hold if we 
replace (P'*'') by an arbitrary probability distribution. This 
provides us a useful tool to study the problem of distribution 
mismatch. In many applications, the true source distribution 
cannot be estimated perfectly. For example, suppose P(a;, y) 
is the true source distribution and Pes{x,y) is the estimated 
source distribution. The initial message is then given by mo = 

iOgp„(a; = l|y)- 

Let PiP (x) (x = 0, 1) be the density of the message from a 
variable node to a check node at the Ith iteration conditioned 
on the event that the variable value is x. Let (Pil'') = P{x = 
O)Pis^(O) + P{x = l)P^i\l) o I-\ The density evolution 
formula of this mismatch problem is 

= (H?) ® A(r-i(p(r((Piri)))))). (5) 

We can use the density evolution formula Q to check whether 
the error probability goes to zero when the distribution mis- 
match occurs. 

We have seen from Example 2 in Section |ll] that under the 
ML-version of the belief-propagation decoding algorithm, the 
Slepian-Wolf coset coding scheme still works even if X is 
nonuniform, as long as Co is a good channel code for chan- 
nel P{y\x) under the belief-propagation decoding algorithm. 
Actually, using the ML-version of the belief-propagation algo- 
rithm for decoding nonuniform X can be viewed as a special 
case of distribution mismatch, where 

PUx = 0\y) ^ P{y\x = 0) 
'°^Pe.(:r = l|y) '°Sp(y|x = l)' 

Thus in this example distribution mismatch does not imply 
decoding failure, but it may cause rate loss. 



IV. Source-Channel Correspondence 

In the density evolution formula Q in channel coding and 
density evolution formula (0} in Slepian-Wolf coding, the 
channel and source statistics come in only through the initial 
message distribution; all the remaining operations depend only 
on the degree distribution. So for a fixed degree distribution 
pair (Ajp), if P^"^ = (P'^"-'), then the two density evolutions 
are completely identical, i.e., we have pC) = (pC)) for all 
I. So a natural question is: For a given Slepian-Wolf initial 
message distribution (P^°-'), does there exist a BIOS channel 
whose initial message distribution p(°) is the same as (p(°')? 
We now proceed to answer this question. 

Definition 2 ( [9], Definition 1): We call a distribution Q 
symmetric if J h{m)Q{dm) = J e~"''h{~m)Q{dm) for any 
function h for which the integral exists. 

Lemma 1: (P(')) IS symmetric. 

Remark: The reason why (P^'^} is symmetric even when 
there is no symmetry in the source distribution P{x. y) is that 
the coset coding scheme is used, and the prior distribution 
P{x) is incorporated in the decoding. 

The following theorem is the main result of this section, 
which essentially says that under belief propagation decoding, 
for all {X, Y) pairs, Slepian-Wolf code design with linear 
codes reduces to design of codes for certain symmetric chan- 
nels. 

Theorem 5: For any source distribution P{x, y) on X x y 
(X = {0,1}, 13^1 < oo) with conditional entropy H{X\Y), 
there exists a unique BIOS channel P'{y'\x') with capacity 
C such that its initial message distribution P'°) is the same 
as the initial message distribution (P'"') induced by P{x,y). 
Furthermore, we have H{X\Y) + C — 1. The conversion from 
P{x,y) to P'{y'\x') is given in Fig. 1. 

Remark: It can be seen from Fig. 1 that P'{y'\x') is in 
general different from P{y\x), and P'{y'\x') is symmetric 
even when P{y\x) is not. Furthermore, if P{x) is nonuniform, 
then P'{y'\x') is different from P{y\x) even when P{y\x) is 
symmetric. 




Fig. 1. Source-to-channel conversion 



Let Ch(-) be the function that maps P{x,y) to P'{y'\x'). 
It turns out this function is not invertible. 

Definition 3 (Equivalence): Two sources distributions, 
P{x,y) and P'{x',y'), are equivalent if they induce 
the same initial message distribution (P^"^) (or if 

Ch(P(x,2/))=Ch(P'(x',y')))- 

For a symmetric distribution Q given by Q (in ^ = 

a-i, i — 0,1, ■■■ ,n — 1, where G (0, 1] and X^iLo^ '^i ~ 
we can compute all source distributions for which the induced 
initial message distribution {P^^^) is equal to Q. These can be 
written (possibly after relabelling) in the following parametric 
form: 

P{y = i) = ai{ai + ttn-i-i), (6) 
P{x = 0\y = i) = , (7) 

Pix^l\y^i) - ,z = 0,--- ,n-l. (8) 

where ai E [0, 1], I — ai — a„_i_i, z 0, 1, • • • ,n — 1. So 
for a fixed initial message distribution with n probability mass 
points, the set of equivalent source distributions has totally [^J 
degrees of freedom. This should be contrasted with the 2n — 2 
degrees of freedom for source distributions (over X x y with 
\X\ = 2 and |3^| — n) under the conditional entropy constraint. 
We can also see that the equivalent source distributions must 
have the same reverse channel P{x\y)\ the freedom comes 
only from P{y). 

Example 3: Consider the following class of source distri- 
butions P{y = e) = e, P{y G {0, 1}) = 1 - e, P{x = i\y 
i) = 1, i = 0, 1, P{x = 0\y = e) = P(x = = e) = i. 
It is easy to verify that this class of source distributions is an 
equivalence class. In fact, for any distribution P{x, y) in this 
class, Ch(P(x, y)) is the binary erasure channel with erasure 
probability e. Since the capacity of the binary erasure channel 
can be achieved with LDPC codes under the belief -propagation 
decoding algorithm, it follows that for distributions in this 
class, the Slepian-Wolf limit is achievable with the LDPC coset 
coding scheme under belief-propagation decoding. 

Example 4: It can be verified that for fixed q, the source 
distributions in Example 1 are all equivalent. 

Given two channels, P{y\x) and P'{y'\x'), we say 
P'{y'\x') < P{y\x) if P'{y'\x') is physically degraded with 
respect to P{y\x). We now generalize this concept to source 
distributions. 

Definition 4 (Monotonicity): Given two source distribu- 
tions, P{x,y) and P'{x',y'), we say P'{x',y') < P{x,y) 
if Ch{P'{x',y'))<Ch{P{x,y)). 

Remark: One may tend to define "monotonicity" in the 
following way: P'{x',y') < P{x,y) if (possibly after re- 
labelling) P'{x) ^ P{x) for all X e X and P'iy'\x') 
is physically degraded with respect to P{y\x). It turns out 
the "monotonicity" in this sense also satisfies the condition 
Ch{P'{x',y')) ^ Ch{P{x,y)), as illustrated in Fig. 2. 

The following theorem follows immediately from the mono- 
tonicity of density evolution with respect to physically de- 
graded channels. 
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good for one BIOS channel is likely to be good for many other 
BIOS channels. Therefore, Fig. 3 is simply a manifestation of 
this phenomenon in the Slepian-Wolf source coding scenario. 



P{x,y) 



Fig. 2. Preservation of source monotonicity in the channel domain 



Theorem 6: Suppose P'{x',y') ^ P{x,y). For any fixed 
degree distribution pair (A,/?), if, for P'{x',y'), the error 
probability under density evolution goes to zero, then it must 
also go to zero for P{x,y). 

Definition 5: For a fixed degree distribution pair (A, p) and 
a class of distributions P over Xxy, the feasible domain T C 
V with respect to (A, p) is the set of distributions P(a;, y) G V 
for which the error probability under density evolution goes 
to zero. 

Example 5: Let V be the class of distributions P{x, y) over 
{0, 1} X {0, 1} such that P{y = l\x = Q) ^ P{y = 0|x = 1). 
Let P{x = 0) = p and P{y = l\x = 0) = P{y = 0\x = 1) = 
q. Then V can be parameterized by p and q with p,q [0, ^]. 

Let S{R) = {{p,q):H{X\Y)\p,,<R,p,qe [0,i]}. For 
any degree distribution pair (A, p) with syndrome rate less than 
or equal to R, by Slepian-Wolf theorem we must have C 
S{R). In Fig. 3 the area below the solid curve is <S(|). We 
also plot the feasible domains of several rate one-half codes. 

1) Code 1: Its degree distribution pair is given in [9, Ex- 
ample 2]. This code is designed for a binary symmetric 
channel, which corresponds to p = i in Fig. 3. The 
feasible domain of this degree distribution pair is the 
area below the "o" curve. 

2) Code 2: A(a;) = 0.234029a; + 0.212425a;2 + 
0.146898a;5 + 0.102840a;S + 0.303808x19, p{x) = 
0.71875a;^ + 0.28125a;S. This code is designed for a 
binary-input AWGN channel [11]. The feasible domain 
is the area below the "H-" curve. 

3) Code 3 is the (3, 6)-regular code. Its feasible domain is 
the area below the "*" curve. 

4) Code 4 is the (4, 8)-regular code. Its feasible domain is 
the area below the "□" curve. 

It can be seen that although code 1 is designed for the case p — 
i, it performs very well over the whole range; the performance 
of code 2 is also quite good, although it is designed for binary- 
input AWGN channel. This should not be too surprising since 
under density evolution, every Slepian-Wolf coding problem 
is equivalent to a channel coding problem for a corresponding 
BIOS channel, and it is a well-known phenomenon that a code 
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Fig. 3. Feasible domain 

V. Conclusion 

We have established an intimate connection between 
Slepian-Wolf coding and channel coding, which clarifies a 
misconception in the area of Slepian-Wolf code design. In- 
terested readers may refer to [12] for more details. 
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